Image processing system using unified images

ABSTRACT

An image processing system has a plurality of cameras and a display that are mounted on a machine. The plurality of cameras are configured to generate image data for an environment of the machine. The image processing system also has a processor connected to the plurality of cameras and the display. The processor is configured to access the image data from the plurality of cameras, access parameters associated with the plurality of cameras, generate a unified image by combining the image data from the plurality of cameras based at least in part on the parameters, access state data associated with the machine, select a portion of the unified image based at least in part on the state data, and render the portion of the unified image on the display.

TECHNICAL FIELD

The present disclosure relates generally to an image processing system,and more particularly, to an image processing system using unifiedimages.

BACKGROUND

Various machines, such as those that are used to dig, loosen, carry,compact, etc., different materials, may be equipped with imageprocessing systems including cameras. The cameras capture images of theenvironment around the machine, and the image processing systems renderthe images on a display within the machine. Such image processingsystems may assist the operators of the machines by increasingvisibility and may be beneficial in applications where the operators'field of view is obstructed by portions of the machine or otherobstacles.

One problem with current image processing systems is that they presentimages from cameras simultaneously as several independent images on onedisplay or on several independent displays. While improving visibility,such systems may not assist operators in operating machines safely. Asimages from all cameras are being shown simultaneously, it may bedifficult for the operator to steer the machine and pay attention to alldisplays concurrently. As a result, an operator may not see an object inthe environment of the machine that has been captured by the imageprocessing system, and a collision may result.

A system that may be used to improve visibility is disclosed in U.S.Patent Application Publication 2012/0262580 to Huebner at al., whichpublished Oct. 18, 2012 (the '580 Publication). The system of the '580Publication provides a surround view from a vehicle by way of cameraspositioned at various locations on the vehicle. The cameras can generateimage data corresponding to the surround view, and a processing devicecan process the image data and generate the surround view on a simulatedpredetermined shape that can be viewed from a display. The simulatedpredetermined shape can have a flat bottom with a rectangular shape anda rim with a parabolic shape. Although the system of the '580Publication may increase visibility, it does not necessarily increasesafety as the entire surround view is displayed.

The disclosed systems and methods are directed to overcoming one or moreof the problems set forth above and/or other problems of the prior art.

SUMMARY

In one aspect, the present disclosure is directed to an image processingsystem including a plurality of cameras and a display mounted on amachine. The plurality of cameras may be configured to generate imagedata for an environment of the machine. The image processing system alsoincludes a processor connected to the plurality of cameras and thedisplay. The processor may be configured to access the image data fromthe plurality of cameras, access parameters associated with theplurality of cameras, generate a unified image by combining the imagedata from the plurality of cameras based at least in part on theparameters, access state data associated with the machine, select aportion of the unified image based at least in part on the state data,and render the selected portion of the unified image on the display.

In another aspect, the present disclosure is directed to a method ofprocessing an image including accessing image data from a plurality ofcameras mounted on a machine, accessing parameters associated with theplurality of cameras, generating a unified image by combining the imagedata from the plurality of cameras based on the parameters, accessingstate data associated with the mobile machine, selecting a portion ofthe unified image data based at least in part on the state data, andrendering the selected portion of the unified image on a display mountedon the machine.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial illustration of an exemplary disclosed machine.

FIG. 2 is a block diagram illustrating an exemplary image processingsystem for the machine of FIG. 1.

FIG. 3 is a pictorial illustration of an exemplary disclosed unifiedimage generated by the image processing system of FIG. 2.

FIG. 4 is a pictorial illustration of several examples of portions ofunified images rendered on several displays by the image processingsystem of FIG. 2.

FIG. 5 is a pictorial illustration of an exemplary disclosed displayshowing a portion of a unified image with overlays that may be renderedby the image processing system of FIG. 2.

FIG. 6 is a flow chart illustrating an exemplary disclosed method thatmay be performed by the image processing system of FIG. 2.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary machine 110 having multiple systems andcomponents that cooperate to accomplish a task. Machine 110 may embody afixed or mobile machine that performs some type of operation associatedwith an industry such as mining, construction, farming, transportation,or any other industry known in the art. For example, machine 110 may bean earth moving machine such as an excavator, a dozer, a loader, abackhoe, a motor grader, a dump truck, or any other earth movingmachine. Machine 110 may include one or more radar devices 120 a-120 hand cameras 140 a-140 d. Radar devices 120 a-120 h and cameras 140 a-140d may be operational on machine 110 during operation of machine 110,e.g., as machine 110 moves about an area to complete certain tasks suchas digging, loosening, carrying, drilling, or compacting differentmaterials.

Machine 110 may use radar devices 120 a-120 h to detect objects in theirrespective fields of view 130 a-130 h. For example, radar device 120 amay be configured to scan an area within field of view 130 a to detectthe presence of one or more objects. During operation, one or moresystems of machine 110 (not shown) may process radar data received fromradar device 120 a to detect objects that are in the environment ofmachine 110. For example, a collision avoidance system may use radardata to control machine 110 to prevent it from colliding with objects inits path. Moreover, one or more systems of machine 110 may generate analert, such as a sound, when an object is detected in the environment ofmachine 110. Cameras 140 a-140 d may be attached to the frame of machine110 at a high vantage point. For example, cameras 140 a-140 d may beattached to the top of the frame of the roof of machine 110. Machine 110may use cameras 140 a-140 d to detect objects in their respective fieldsof view. For example, cameras 140 a-140 d may be configured to recordimage data such as video or still images.

During operation, one or more systems of machine 110 (not shown) mayrender the image data on a display of machine 110 and/or may process theimage data received from the cameras to detect objects that are in theenvironment of machine 110. For example, when the one or more systems ofmachine 110 detect an object in the image data, the image data may berendered on the display. According to some embodiments, the one or moresystems of machine 110 may render an indication of the location of thedetected object within the image data. For example, the one or moresystems of machine 110 may render a colored box around the detectedobject, or render text below, above, or to the side of the detectedobject.

While machine 110 is shown having eight radar devices 120 a-120 h, andfour cameras 140 a-140 d, those skilled in the art will appreciate thatmachine 110 may include any number of radar devices and cameras arrangedin any manner. For example, machine 110 may include four radar deviceson each side of machine 110.

FIG. 2 is a block diagram illustrating an exemplary image processingsystem 200 that may by installed on machine 110 to capture and processimage data in the environment of machine 110. Image processing system200 may include one or more modules that when combined perform imageprocessing, image rendering, and object detection. For example, asillustrated in FIG. 2, image processing system 200 may include radarinterface 205, camera interface 206, machine interface 207, objectdetector 215, alert processor 250, and image processor 255. While FIG. 2shows the components of image processing system 200 as separate blocks,those skilled in the art will appreciate that the functionalitydescribed below with respect to one component may be performed byanother component, or that the functionality of one component may beperformed by two or more components. For example, the functionality ofalert processor 250 may be performed by object detector 215 or thefunctionality of image processor 255 may be performed by two components.

According to some embodiments, the modules of image processing system200 may include logic embodied as hardware, firmware, or a collection ofsoftware written in a programming language. The modules of imageprocessing system 200 may be stored in any type of computer-readablemedium, such as a memory device (e.g., random access, flash memory, andthe like), an optical medium (e.g., a CD, DVD, BluRay®, and the like),firmware (e.g., an EPROM), or any other storage medium. The modules maybe configured for execution by one or more processors to cause imageprocessing system 200 to perform particular operations. The modules ofthe image processing system 200 may also be embodied as hardware modulesand may be comprised of connected logic units, such as gates andflip-flops, and/or may be comprised of programmable units, such asprogrammable gate arrays or processors, for example.

Image processing system 200 may include radar device 120 and camera 140.Radar device 120 may correspond to one or more of radar devices 120a-120 h and camera 140 may correspond to one or more of cameras 140a-140 d, for example. Moreover, while only one radar device 120 and onecamera 140 are shown in FIG. 2, those skilled in the art will appreciatethat any number of radar devices and cameras may be included in imageprocessing system 200.

In some aspects, before the image processing system 200 can processradar data from radar device 120 and image data from camera 140, theradar data and the image data must be converted to a format that isconsumable by the modules of image processing system 200. Accordingly,radar device 120 may be connected to radar interface 205, and camera 140may be connected to camera interface 206. Radar interface 205 and camerainterface 206 may receive analog signals from their respective devicesand convert them to digital signals which may be processed by the othermodules of image processing system 200. For example, radar interface 205may create digital radar data using information it receives from radardevice 120, and camera interface 206 may create digital image data usinginformation it receives from camera 140. According to some embodiments,radar interface 205 and camera interface 206 may package the digitaldata in a data package or data structure along with metadata related tothe converted digital data. For example, radar interface 205 may createa data structure or data package that has metadata and a payload. Thepayload may represent the radar data from radar device 120.Non-exhaustive examples of data included in the metadata related to theradar data may include the orientation of radar device 120, the positionof radar device 120, and/or a time stamp for when the radar data wasrecorded. Similarly, camera interface 206 may create a data structure ordata package that has metadata and a payload representing image datafrom camera 140. The metadata related to the image data may includeparameters associated with camera 140 that captured the image data.Non-exhaustive examples of the parameters associated with camera 140 mayinclude the orientation of camera 140, the position of camera 140 withrespect to machine 110, the down-vector of camera 140, the range of thecamera's field of view, a priority for image processing associated withcamera 140, and a time stamp for when the image data was recorded.Parameters associated with the camera may be stored in a configurationfile, database, data store, or some other computer readable mediumaccessible by camera interface 206. The parameters may be set by anoperator prior to operation of machine 110.

In some embodiments, radar device 120 and camera 140 may be digitaldevices that produce digital data, and radar interface 205 and camerainterface 206 may package the digital data into a data structure forconsumption by the other modules of image processing system 200. Radarinterface 205 and camera interface 206 may expose an application programinterface (API) that exposes one or more function calls allowing theother modules of image processing system 200, such as object detector215, to access the radar data and the image data.

In addition to radar interface 205 and camera interface 206, imageprocessing system 200 may also include machine interface 207. Machineinterface 207 may connect with one or more sensors deployed on machine110 and may translate signals from the one or more sensors to digitaldata that may be consumed by the modules of image processing system 200.The digital data may include state data that includes informationrelated to machine's 110 current operation. For example, the state datamay include the current speed of machine 110, the current direction ofmachine 110 (e.g., forward or backward), the current steering angle ofmachine 110, or the acceleration of machine 110. The state data may alsoinclude information about tools or other work components of machine 110.For example, the state data may include the position of loading ordigging arms, or the angle/position of a load bed attached to machine110. The state data may also include metadata such as a time stamp or anidentifier of the tool or work component to which the state dataapplies. Machine interface 207 may expose an API providing access to thestate data of the machine 110 to the modules of image processing system200, such as alert processor 250 and object detector 215.

Image processing system 200 may also include object detector 215. Objectdetector 215 accesses data from the radar interface 205 and the camerainterface 206 and processes it to detect objects that are in theenvironment of machine 110. The radar data accessed from radar interface205 may include an indication that an object was detected in theenvironment of the machine 110. Object detector 215 may access radardata by periodically polling radar interface 205 for radar data andanalyzing the data to determine if the data indicates the presence of anobject. Object detector 215 may also access radar data through an eventor interrupt triggered by radar interface 205. For example, when radardevice 120 detects an object, it may generate a signal that is receivedby radar interface 205, and radar interface 205 may publish an event toits API indicating that radar device 120 has detected an object. Objectdetector 215, having registered for the event through the API of radarinterface 205, may receive the radar data and analyze the payload of theradar data to determine whether an object has been detected.

As processing image data is computationally expensive, object detector215 may advantageously limit the amount of image data that is processedby using radar data corresponding to the image data. The radar data maybe used, for example, to limit processing to the parts of the image datawhere an object is expected. For example, object detector 215 may mapaccessed radar data to accessed image data and only process the portionsof the image data that correspond to an object detected in the accessedradar data. Object detector 215 may map radar data to image data usingmetadata related to the orientation and position of radar device 120 andcamera 140. For example, when object detector 215 receives radar datafrom radar device 120 positioned on the rear of machine 110, it may mapthat radar data to image data from camera 140 that is also positioned onthe rear of machine 110.

In addition to the orientation and position of radar device 120, theradar data may indicate a location within radar device's 120 field ofview 130 where the object was detected. For example, the radar data mayindicate the distance and angular position of the detected object. Insome embodiments, object detector 215 may map the distance and angularposition of the object in the radar data to a pixel location in theimage data. The mapping may be accomplished through a look-up tablewhere distances and angular positions for radar device 120 are linked topixels of the images captured by camera 140. For example, a point at 5meters, 25 degrees in radar device's 120 field of view may correspond toa pixel at (300, 450) in an image captured by camera 140. In someembodiments, radar interface 205 may map radar data to image data andthe payload of the radar data may be expressed in pixels, as opposed todistance and angular position. The look-up table may be stored in acomputer readable data store or configuration file that is accessible byobject detector 215 or radar interface 205, and the look-up table may beconfigurable based on the position of each radar device and camera onmachine 110 and the application of machine 110. Although a look-up tableis one method by which object detector 215 or radar interface 205 maymap radar data to image data, those skilled in the relevant art willappreciate that other methods for mapping radar data to image data maybe used to achieve the same effect.

Object detector 215 may also process image data to detect objects withinthe image data. As indicated above, object detector 215 may only processa portion of the image data that has been mapped to radar dataindicating the presence of an object. Object detector 215 may detectobjects in the image by using edge detection techniques. For example,the object detector 215 may analyze the mapped image data for placeswhere image brightness changes sharply or has discontinuities. Objectdetector 215 may employ a known edge detection technique such as a Cannyedge detector. Although edge detection is one method by which objectdetector 215 may detect objects in images, those skilled in the relevantart will appreciate that other methods for detecting objects in imagedata may be used to achieve the same effect.

Although the exemplary object detector 215 of FIG. 2 uses radar data andimage data to detect objects in the environment of machine 110, objectdetector 215 may use other object detection techniques known by thosewith skill in the art. For example, object detector 215 may use soundnavigation and ranging (SONAR), light detection and ranging (LIDAR),radio-frequency identification (RFID) or global position satellite (GPS)data to detect objects in the environment of machine 110.

Image processing system 200 may also include alert processor 250. Oncean object has been detected, alert processor 250 may analyze the objectand state data received from machine interface 207 to determine if analert needs to be generated. Alerts may be generated when a collision islikely to occur between the detected object and machine 110. Alerts mayvary depending when the collision is likely to occur. For example, thealert processor 250 may generate a first alert that displays anindication of a detected object on a display 260 as soon as objectdetector 215 detects an object, but alert processor 250 may generate asecond alert that makes a sound and flashes a warning when a detectedobject is about to collide with machine 110.

Alert processor 250 advantageously uses the state data of machine 110 incombination with detected object data to determine whether to generatean alert. Alert processor 250 may use the speed and direction of machine110, obtained from machine interface 207, to determine the likely pathof machine 110. After determining the likely path, alert processor 250may determine whether any detected objects are in the likely path, andit may generate an appropriate alert, if necessary. For example, alertprocessor 250 may determine that machine 110 is moving along a straightpath and that a detected object is along that straight path. Alertprocessor 250 may determine that if machine 110 does not changedirection and if the detected object does not move, a collision islikely to occur in 10 seconds. Accordingly, alert processor 250 maygenerate an alert such as an audible warning. Alert processor 250 mayalso render a visual warning on display 260.

Image processing system 200 may include image processor 255. Imageprocessor 255 may manipulate image data received from camera interface206 and/or process the image data. Image processor 255 may, for example,combine the image data captured by cameras 140 a-140 d into a unifiedimage. Image processor 255 may also select a portion of the unifiedimage to render on display 260 and may generate overlays conveyingcontextual information on display 260. Display 260 is typically disposedin close proximity to the cabin of machine 110 and within the view ofthe operator of machine 110. Display 260 may be any display capable ofrendering graphics generated by a general purpose computing system. Forexample, display 260 may be a LCD screen, LED screen, CRT screen, plasmascreen, or some other screen suitable for use in machine 110. Display260 may be connected to the processor of image processing system 200,and the processor may execute instructions to render graphics and imageson display 260. The functionality of image processor 255 will now bediscussed in greater detail with reference made to FIGS. 3-5.

In some embodiments, image processor 255 may combine image data capturedby cameras 140 a-140 d into unified image 320. FIG. 3 is a pictorialillustration of one example of image processor 255 combining sourceimage data 310 a-310 d to generate unified image 320. Unified image 320may represent all image data available for the environment of machine110. Conceptually, unified image 320 represents a 360-degree view of theenvironment of machine 110, with machine 110 at the center of the360-degree view. According to some embodiments, unified image 320 may bea non-rectangular shape. For example, unified image 320 may behemispherical and machine 110 may be conceptually located at the pole,and in the interior, of the hemisphere.

Image processor 255 may generate unified image 320 by mapping pixels ofsource image data 310 a-310 d captured by cameras 140 a-140 d to a pixelmap. The pixel map may be divided into sections, with each sectioncorresponding to one of source image data 310 a-310 d. For example, asshown in FIG. 3, first camera 140 a captures source image data 310 awhich is mapped to section 331, second camera 140 b captures sourceimage data 310 b which is mapped to section 332, third camera 140 ccaptures source image data 310 c which is mapped to section 333, andfourth camera 140 d captures source image data 310 d which is mapped tosection 334. Pixels may be mapped directly using a one-to-one orone-to-many correspondence, and the mapping may correlate a twodimensional point from source image data 310 a-310 d to a threedimensional point on the map used to generate unified image 320. Forexample, a pixel of source image data 310 a located at (1,1) may bemapped to location (500, 500, 1) of unified image. The mapping may beaccomplished using a look-up table that may be stored in a computerreadable data store or configuration file that is accessible by imageprocessor 255. The look-up table may be configured based on the positionand orientation of each camera on machine 110. Although a look-up tableis one method by which image processor may map radar source image data310 a-310 d to unified image 320, those skilled in the relevant art willappreciate that other methods for mapping radar data to image data maybe used to achieve the same effect.

Image processor 255 may also use the parameters associated with cameras140 a-140 d to map pixels from source image data 310 a-310 d to unifiedimage 320. The parameters may be included in the metadata of sourceimage data 310 a-310 d. For example, the parameters may include theposition of each camera 140 a-140 d with respect to machine 110. Imageprocessor 255 may correlate sections 331, 332, 333, 334 of unified image320 with machine 110, and image processor 255 may use the correlationsto determine which of source image data 310 a-310 d to map to eachsection. For example, image processor 255 may correlate section 331 withthe front of machine 110. When image processor receives source imagedata 310 a, the parameters included in the metadata associated withsource image data 310 a may indicate that it was captured by camera 140a. The parameters may also indicate that camera 140 a is positioned onthe front of machine 110. Image processor 255 analyzes the parametersand determines that source image data 310 a should be mapped to section331. Thus, as image processor 255 accesses source image data 310 a-310d, it can correctly map it to sections 331, 332, 333, 334 of unifiedimage 320.

The parameters associated with cameras 140 a-140 d may also includeranges of fields of view for each of cameras 140 a-140 d. The ranges mayinclude the boundaries of the image with respect to machine 110expressed as an angle from the down-vector of the camera and a distance.For example, camera 140 b may capture images at a sixty-five degreeangle from its down-vector for three hundred meters. As the ranges offields of view of adjacent cameras may overlap, image processor 255 mayuse the ranges of fields of view to determine the amount of overlap. Forexample, camera 140 b may be attached to the left side of machine 110and camera 140 c may be attached to the back of machine 110. The leftside of the image captured by camera 140 b may overlap with the rightside of the image captured by camera 140 c. Image processor 255 may usethe parameters associated with camera 140 b and camera 140 c todetermine how much overlap exists in the image data.

When multiple pieces of source image data overlap, a conflict arises asto which source image data will be used in unified image 320. In oneembodiment, image processor 255 may resolve the conflict by using theparameters associated with the camera that captured the source imagedata, specifically, the priorities of the cameras for image processing.For example, image processor 255 may receive source image data 310 cfrom camera 140 c and source image data 310 d from camera 140 d. Camera140 c may correspond to the right side of machine 110 and camera 140 dmay correspond with the rear of machine 110. As a result, source imagedata 310 c and source image data 310 d may include some overlap. Imageprocessor 255 may refer to the parameters associated with camera 140 cand 140 d and determine that camera 140 d has a higher priority forimage processing than camera 140 c. Accordingly, image processor 255 maymap pixels from source image data 310 d to unified image 320 for theoverlapping portion.

In some embodiments, the parameters associated with cameras 140 a-140 dmay specify the mapping image processor 255 uses to create unified image320. The mapping may be part of the configuration of cameras 140 a-140 dand may be stored in a configuration file, database, data store, or someother computer readable medium. For example, when source image data 310a is generated (for example, by camera interface 206), the mapping ofthe pixels of source image data 310 a to the pixels of the map used togenerate unified image 320 may be included in the parameters of themetadata for source image data 310 a. Thus, when image processor 255receives source image data 310 a, it will simply map the pixels fromsource image data 310 a to unified image 320 according to the mappingspecified in the parameters. Such an arrangement may be beneficial whereresources available to image processor 255 are limited.

According to some embodiments, image processor 255 may select a portionof the unified image 320 for rendering on display 260. The portion maybe selected using viewpoint 350. Conceptually, viewpoint 350 representsa plane from which unified image 320 may be viewed, and the pixelslocated under the plane form the portion of unified image 320 that imageprocessor 255 renders on display 260. For example, as shown in FIG. 3,viewpoint 350 is above all of unified image 320, and all of the pixelsof unified image 320 are under viewpoint 350. As a result, imageprocessor 255 selects all of the pixels of unified image 320 forrendering on display 260 resulting in display image 360.

In some embodiments, viewpoint 350 is rectangular and moves aroundunified image 320 depending on the state data of machine 110 and/orwhether object detector 215 has detected any objects in the environmentof machine 110. FIG. 4 illustrates three examples of portions of unifiedimage 320 rendered on display 260. The portions rendered on display 260vary depending on the position of viewpoint 350. The position ofviewpoint 350 may be affected by the motion of machine 110, and whetheran object has been detected in the environment of machine 110. In FIG.4, the three examples are denoted by the letters A, B, and C, withelements having A following their number (such as machine 110 a, display260 a, unified image 320 a, viewpoint 350 a, vehicle 410 a, worker 420a, and rendered image 450 a) corresponding to the first example,elements having B following their number corresponding to the secondexample, and elements having C following their number corresponding tothe third example.

In Example A, machine 110 a is moving forward. In front of machine 110 ais vehicle 410 a, and worker 420 a is behind machine 110 a. As machine110 a is moving forward, image processor 255 may access state dataassociated with machine 110 a indicating forward motion. Image processor255 may adjust viewpoint 350 a toward the portion of unified image 320 acorresponding with the front of machine 110 a, that is, viewpoint 350 amoves to the front of unified image 320 a. Image processor 255 selects aportion of unified image 320 a to render on display 260 a by selectingthe pixels of unified image 320 a that are under the plane of viewpoint350 a. Image processor 255 may render the portion of unified image 320 aon display 260 a as rendered image 450 a. Rendered image 450 a includesvehicle 410 a as it is in front of machine 110 a.

In Example B, machine 110 b is moving backward. In front of machine 110b is vehicle 410 b, and worker 420 b is behind machine 110 b. As machine110 b is moving backward, image processor 255 may access state dataassociated with machine 110 b indicating backward motion. Imageprocessor 255 may adjust viewpoint 350 b toward the portion of unifiedimage 320 b corresponding with the back of machine 110 b, that is, theback of unified image 320 b. Image processor 255 selects a portion ofunified image 320 b to render on display 260 b by selecting the pixelsof unified image 320 b under the plane of viewpoint 320 b. Imageprocessor 255 may render the portion of unified image 320 b on display260 b as rendered image 450 b. Rendered image 450 b includes worker 420b as he is behind machine 110 b.

Example C provides an example of image processor 255 using state data ofmachine 110 c and object detector 215 to adjust viewpoint 350 c. InExample C, machine 110 c is moving forward. Vehicle 410 c is in front ofmachine 110 c, and worker 420 c is behind machine 110 c. Image processor255 may access state data of machine 110 c that indicates forwardmotion. Image processor 255 may also receive data from object detector215 indicating that an object, worker 420 c, was detected in field ofview 130 of a radar device located on the rear of machine 110 c. Imageprocessor 255 may then adjust viewpoint 350 c to encompass the portionsof unified image 320 c corresponding to both the front and rear ofmachine 110 c, that is, the front and rear of unified image 320 c. Imageprocessor 255 may select the portion of unified image 320 c to render ondisplay 260 c by selecting the pixels of unified image 320 c underviewpoint 350 c. Image processor 255 may render the portion of unifiedimage 320 c on display 260 c as rendered image 450 c. Rendered image 450c includes vehicle 410 c as it is the direction of motion of machine 110c and worker 420 c as he was detected by object detector 215.

In some embodiments, as machine 110 traverses a work site, viewpoint 350will adjust in response to the motion of machine 110. In addition, asobjects enter the environment of machine 110 and are detected by objectdetector 215, viewpoint 350 may adjust to encompass the portions ofunified image 320 corresponding to the detected object. Thus, at anygiven time, viewpoint 350 may be above any combination of source imagedata 310 a-310 d from cameras 140 a-140 d, and image processor 255 mayselect the corresponding portion of unified image 320 to render ondisplay 260. As a result, image processor 255 advantageously offers anoperator a single rendered image encompassing the portions of theenvironment of machine 110 that may be of interest to the operator.Thus, image processing system 200 provides awareness of those portionsof the environment to the operator of machine 110, and the operator mayoperate machine 110 more safely.

In some embodiments, image processing system 200 may include displaycontrols 261 (shown in FIG. 2). The operator may use display controls261 to input his preference for viewing an area of the environment ofthe machine. Image processor 255 may render a portion of the unifiedimage on display 260 based on the inputs received from the operatorusing display controls 261. In some embodiments, the display controls261 may include buttons that allow an operator to select the portion ofthe unified image that image processing system 200 renders on thedisplay. The buttons may be physical buttons located near display 260,or when display 260 is a touch screen, the buttons may be virtualbuttons rendered by image processing system 200 on the touch screen.Using display controls 261, an operator may input his preference for theviewing area by expanding or shifting the portion of unified image 320that is rendered on display 260. For example, image processing system200 may render a portion of the unified image corresponding with thefront of machine 110 as the machine 110 moves forward. The operator mayuse the display controls 261 to shift the portion to the left of machine110. In some embodiments, display controls 261 may be used to displaythe entire unified image on display 260. For example, an operator mayselect a button providing a bird's-eye-view of the unified image,thereby showing the operator all image data from cameras 140 a-140 d atthe same time.

In some embodiments, image processor 255 may generate one or moreoverlays to render on display 260. The overlays may provide informationto the operator of machine 110 that helps the operator operate machine110 more safely or more effectively. The overlays may include, forexample, the projected path of machine 110 based on state data. Imageprocessor 255 may determine the projected path of machine 110 based onstate data accessed from machine interface 207. The state data mayinclude, for example, the speed of machine 110 and the steering angle ofmachine 110. Image processor may use the speed and steering angle tocreate a predicted path. The predicted path may be mapped to unifiedimage 320 and rendered on display 260.

FIG. 5 is a pictorial illustration of one example of display 260 showinga portion of unified image 320 rendered on display 260 with target pathoverlay 540 and projected path overlay 545. In FIG. 5, machine 110 isbacking into tunnel 510. As machine is moving backward, the state dataassociated with machine 110 may indicate backward motion. Imageprocessor 255 may adjust viewpoint 350 to cover the portion of unifiedimage 320 corresponding with the back of machine 110, and may render theportion of unified image 320 under viewpoint 350 as rendered image 520.Image processor 255 may also determine a predicted path using the statedata associated with machine 110. The state data may indicate thatmachine 110 has a steering angle that is not straight and curvingslightly to the right. As a result, image processor 255 may generateprojected path overlay 545 showing the predicted path of machine 110 andrender projected path overlay 545 on display 260.

Image processor 255 may also, in some embodiments, determine a targetpath for machine 110. The target path may be the best path from thecurrent position of machine 110 to a destination target. Image processor255 may determine the target path by first determining a straight pathfrom machine 110 to a destination target. If the unified image 320and/or radar data indicate the straight path is unobstructed, imageprocessor 255 may render target path overlay 540 on display 260. Forexample, as shown in FIG. 5, destination target 530 is inside tunnel510. Image processor 255 may have determined that the straight path tothe destination target 530 was unobstructed, so it rendered target pathoverlay 540 on display 260. When image processor 255 determines thetarget path is obstructed, it may adjust the target path to circumventany obstacles.

In some embodiments, the destination target may be a GPS coordinate thatimage processor 255 maps to unified image 320 based on the position ofmachine 110 and the position of the destination target. The destinationtarget may also be a physical marker with specific characteristics thatare detectable by object detector 215. For example, the physical markermay be a cone, barrel or flag with a unique shape known to objectdetector 215. In some embodiments, the operator of machine 110 can usedisplay controls 261 to set the destination target that may be used byimage processor 255 to determine the target path. In some embodiments,display 260 may include a touch screen and an operator may set adestination target by touching the touch screen at the location wherethe operator wants to place the destination target. In some embodiments,display controls 261 may include a joystick, mouse, light pen, arrowkeys, or other directional control allowing the operator to manipulate auser interface element on display 260. After the operator moves the userinterface element to the location for the destination target, she maypress a button selecting that location as the destination target.

INDUSTRIAL APPLICABILITY

The disclosed image processing system 200 may be applicable to anymachine that includes cameras. The disclosed image processing system 200may enhance operator awareness by rendering a portion of a unified imageon a display and selecting the rendered portion based on state dataassociated with machine 110 or objects detected in the environment ormachine 110. The operation of image processing system 200 will now beexplained.

FIG. 6 is a flow chart illustrating method 600 that may be performed byimage processing system 200. During the operation of machine 110, imageprocessing system 200 may perform method 600 by first accessing imagedata captured by cameras 140 a-140 d, at step 610. For example, imageprocessing system 200 may access the image data of cameras 140 a-140 dthrough camera interface 206. The accessed image data may include apayload including a series of images captured by cameras 140 a-140 d, aswell as metadata including parameters associated with each of cameras140 a-140 d. The parameters may describe the orientation of the camera,the position of the camera with respect to machine 110, the down-vectorof camera, and the range of the camera's field of view. For example, theparameters may indicate that a camera is located on the front of machine110, or that the range of the camera's field of view is forty-fivedegrees from the down-vector of the camera. In some embodiments, therange of the camera's field of view may be expressed in pixel locationscorresponding to the pixel map used to generate the unified image. Theparameters associated with each camera may be used to generate a unifiedimage at step 620.

Image processing system 200 may generate the unified image by combiningthe image data accessed from cameras 140 a-d. Image processing system200 may use a pixel map that maps pixels from the image data to pixellocations on a hemi-spherical pixel map which is used to generate theunified image, as described with respect to FIG. 3 above. The parametersmay specify where the pixels from the image data are placed within thepixel map of the unified image. For example, image processing system 200may map image data captured by a camera positioned on the front ofmachine 110 to the front of the unified image, image data captured by acamera positioned on the right side of machine 110 to the right side ofthe unified image, image data captured by a camera positioned on theleft side of machine 110 to the left side of the unified image, andimage data captured by a camera on the rear of machine 110 to the rearof the unified image. Image processing system 200 may also use theranges of the cameras' fields of view to generate the unified image. Itmay use the ranges to prevent duplicates in the unified image, or toresolve conflicts when more than one pixel from the image data maps to apixel location on the pixel map used to generate the unified image. Forexample, when the camera on the front of machine 110 and the camera onthe right side of machine 110 have overlapping fields of view, imageprocessing system 200 may select the camera on the front of machine 110to provide the pixels in the unified image for the overlapping area.Image processing system 200 may also use the ranges of the fields ofview to generate black-space or white-space pixels corresponding toregions in the environment of machine 110 that have not been captured byany of cameras 140 a-140 d.

Once image processing system 200 has generated the unified image, it mayselect a portion of the unified image to render on display 260. In someembodiments, image processing system 200 selects the portion to renderbased on state data associated with machine 110. At step 630, imageprocessing system 200 accesses the state data associated with machine110, which it may use to determine the portion of the unified image itwill render on display 260. Image processing system 200 may access thestate data from machine interface 207. The state data may include thedirection of motion for machine 110, and image processing system 200 mayuse the direction of motion to select the portion of the unified imageto render. For example, when machine 110 is moving forward, the statedata may indicate machine 110 is moving forward. Image processing system200 may select a portion of the unified image corresponding to the frontof machine 110 to render on display 260. The selected portion may notinclude any image data received from cameras on the rear of machine 110,but the selected portion may include some of the image data from thecameras located on the left and right of machine 110. The state data mayalso indicate that machine 110 is not moving. When the state dataindicates that machine 110 is not moving, image processing system 200may select all of the unified image to render on display, that is, theselected portion of the unified image is the entire unified image.

In some embodiments, image processing system 200 may select the portionof the unified image to render on display 260 based on whether anyobjects are in the environment of machine 110. Accordingly, imageprocessing system 200 may include radar devices 120 a-120 h to detectobjects. At step 640, image processing system 200 accesses radar datafrom radar devices 120 a-120 h, and at step 650, it may analyze theradar data to detect objects. In some embodiments, image processingsystem 200 may analyze image data to detect objects. When an object isdetected, image processing system 200 may select a portion of theunified image to display such that a detected object is included in theportion. For example, if image processing system 200 detects an objecton the right side of machine 110, the portion of the unified imageselected for rendering on display 260 would include the right side ofthe unified image. When the state data of machine 110 indicates motionand image processing system 200 detects an object, image processingsystem 200 may select a portion of the unified image to render thatincludes both the environment that is in the direction of motion and thedetected object. For example, when the state data indicates machine 110is backing up and image processing system 200 detects an object on theleft side of machine 110, the portion of the unified image selected torender on display 260 may include both the rear of machine 110 and theleft side of machine 110.

At step 660, image processing system 200 renders the selected portion ofthe unified image on display 260. Image processing system 200 may also,at step 670, render overlays on top of the rendered portion of theunified image. For example, image processing system 200 may render anoverlay showing a predicted path of machine 110 that it has determinedbased on the state data associated with machine 110. Image processingsystem 200 may also render a target position and a determined targetpath that may assist an operator of machine 110 direct it to the properlocation for completing a task. Overlays may also include, for example,warnings regarding objects in the path of machine 110 that may cause acollision or indications reflecting detected objects in the environmentof machine 110.

Several advantages over the prior art may be associated with the imageprocessing system 200. For example, image processing system 200 providesenhanced awareness to operators of machine 110 by creating a unifiedimage and rendering a portion of the unified image on display 260 withinmachine 110. In addition, image processing system 200 may access statedata associated with machine 110 and use the state data to select theportion of the unified image it renders on display 260 within machine110. Image processing system 200 may also utilize object detector 215configured to detect objects within the environment of machine 110, andit may select the portion of the unified image to display such that anydetected object is included within the selected portion.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed imageprocessing system. Other embodiments will be apparent to those skilledin the art from consideration of the specification and practice of thedisclosed image processing system. It is intended that the specificationand examples be considered as exemplary only, with a true scope beingindicated by the following claims and their equivalents.

What is claimed is:
 1. An image processing system comprising: a plurality of cameras mounted on a machine, the plurality of cameras configured to generate image data for an environment of the machine; a display mounted on the machine; a processor connected to the plurality of cameras and the display, the processor configured to: access the image data from the plurality of cameras; access parameters associated with the plurality of cameras; generate a unified image by combining the image data from the plurality of cameras based at least in part on the parameters; access state data associated with the machine; select a portion of the unified image based at least in part on the state data; and render the selected portion of the unified image on the display.
 2. The image processing system of claim 1, wherein the state data includes a direction of the machine's motion and wherein selecting the portion of the unified image is based at least in part on the direction of the machine's motion.
 3. The image processing system of claim 1, wherein the state data includes an indication that the machine is not in motion and wherein the selected portion of the unified image includes all of the unified image.
 4. The image processing system claim 1, wherein: the processor is further configured to detect an object in the environment of the machine; and the selected portion of the unified image includes the detected object.
 5. The image processing system of claim 4, wherein the processor is further configured to generate an alert when a collision is likely to occur between the detected object and the machine.
 6. The image processing system of claim 1, further comprising: a radar device mounted to the machine and configured to generate radar data; and wherein the processor is further configured to: access the radar data from the radar device; and, detect an object in the accessed radar data, and wherein the selected portion of the unified image includes the detected object.
 7. The image processing system of claim 1, wherein the parameters include a position of the plurality of cameras with respect to the machine.
 8. The image processing system of claim 1, wherein the unified image data is mapped to a hemi-spherical image map.
 9. The image processing system of claim 1, wherein the selected portion of the unified image is rectangular.
 10. The image processing system of claim 1, wherein: the processor is further configured to detect an input describing an operator preference for viewing an area of the environment of the machine, and the selected portion of the unified image includes the area of the environment.
 11. A method for displaying an image comprising: accessing image data from a plurality of cameras mounted on a machine; accessing parameters associated with the plurality of cameras; generating a unified image by combining the image data from the plurality of cameras based on the parameters; accessing state data associated with the machine; selecting a portion of the unified image data based at least in part on the state data; and rendering the selected portion of the unified image on a display mounted on the machine.
 12. The method of claim 11, wherein the state data includes a direction of the machine's motion and wherein selecting the portion of the unified image is based at least in part on the direction of the machine's motion.
 13. The method of claim 11, wherein the state data includes an indication that the machine is not in motion and wherein the selected portion of the unified image includes all of the unified image.
 14. The method of claim 11 further including detecting an object in an environment of the machine and wherein the selected portion of the unified image includes the detected object.
 15. The method of claim 11, further including: accessing radar data from a radar device mounted on the machine; and detecting an object an environment of the machine using the radar data, wherein the selected portion of the unified image includes the detected object.
 16. The method of claim 11, wherein the parameters include a position of the plurality of cameras with respect to the machine.
 17. The method of claim 11, wherein the unified image data is mapped to a hemi-spherical image map.
 18. The method of claim 11 wherein the selected portion of the unified image is rectangular.
 19. The method of claim 11, further including: detecting an input describing an operator preference for viewing an area of an environment of the machine and wherein the selected portion of the unified image includes the area of the environment.
 20. A machine comprising: a cabin, a display disposed within the cabin; a frame; a plurality of cameras connected to the frame, the plurality of cameras configured to generate image data for an environment of the machine; a processor in communication with the plurality of cameras and the display, the processor configured to: access the image data from the plurality of cameras; access parameters for the plurality of cameras; generate a unified image by combining, based on the parameters, the image data from the plurality of cameras; detect an object in the environment of the machine; access state data describing associated with the machine, the state data including a direction of the machine's motion; select a portion of the unified image based at least in part on the state data and a location of the detected object; and, render the selected portion of the unified image on the display. 